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Abstract 

Open Polytechnic is a single-mode provider of distance education with a rich history of 
print-based provision. Strategically, the institution is rapidly adopting an online-only 
approach, with some exceptions for programmes that require student contact. A recent and 
internal review of Moodle, the Open Polytechnic’s learning management system (LMS) 
indicated concern about the likely student experience for an online-only format, so an 
internal system called iQualify was developed. iQualify was developed specifically from a 
user-experience (UX) perspective to optimise the online-only learning experience. As part of 
the drive towards online-only education, Open Polytechnic courses are being purposefully 
redeveloped for iQualify, moving away from print-and textbook-based materials, and 
towards textbook-independent and online-only study. This article uses a UX perspective to 
compare student outcomes, high-level course evaluations, and student perceptions of the two 
approaches—Moodle, with print and textbook; and iQualify online-only. Findings indicate 
that while students tend to prefer printed materials, actual withdrawal and pass rates are not 
affected by an online-only approach. Respondents also indicated a significant preference for 
the new iQualify platform over Moodle, and openness to online-only study. 

Keywords: distance education; iQualify; LMS evaluation; Moodle; online education; Open 
Polytechnic; student preferences; user experience; UX 


Background 

Open Polytechnic is the largest distance learning provider in New Zealand. Each year it provides 
more than 30,000 adult learners with the opportunity to study for formal qualifications. Since 
being established as the Technical Correspondence School in 1946, the institution has practised 
an industrialised approach 1 (Peters, 2007) to course development and student teaching. Open 
Polytechnic now offers a suite of degree programmes alongside its more traditional certificates 
and diplomas; all of these course materials are developed by a team of specialists and taught by 
qualified subject matter experts. Open Polytechnic could be defined as a resource-based, rather 
than lecture-based, institution (Seelig & Nichols, in press). The programmes offered by Open 
Polytechnic lead to a variety of qualifications in business, social work, early childhood 
education, library studies, architectural technology, foundation studies, and other discipline 
areas. 

Open Polytechnic was an early adopter of online education. It implemented the Moodle learning 
management system (LMS) in the mid-2000s, and received a Mellon Award in 2007 in 
recognition of its enthusiastic contribution to Moodle’s development. But until recently, Moodle 
predominantly served to complement printed resources. Although Moodle added a flavour of 
social constructivism to otherwise independent learning resources by encouraging students to 


' An industrialised approach is characterised by a division of labour and centrally coordinated activity. 
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participate in online forums, the actual online course materials were usually in the form of PDF 
versions of printed materials that were previously made available to students. With Moodle in 
place. Open Polytechnic could move toward online assignment submission and marking, and 
took its first forays into developing courses specifically for online-only delivery. 

As outlined in Seelig & Nichols (in press), Corbeil and Corbeil suggest that the 1990s might be 
described as “the era of the learning management system” (2015, sec. 4, para. 1), and popular 
learning management systems developed before the new millennium continue to dominate online 
learning. Although these systems have continued to evolve, they suffer from three distinct 
problems from the perspective of online learning systems. These systems: 

• tend to be based on older technologies and approaches 

• assume that the teacher is solely responsible for the learning experience, and is using a 
lecture-based model 

• assume that online learning should involve formal online collaboration and 
communication. 

As Open Polytechnic deepened its understanding of the online learning activities that would best 
suit its needs as a distance education provider reflecting an industrialised approach, it became 
clear that a traditional LMS was unsuitable because these systems were not developing in ways 
that enhance a genuine, online-only student experience. Subsequently, following an investigation 
of LMS alternatives in late 2013, Open Polytechnic began to explore bespoke possibilities, and 
development of iQualify began in 2014. Rather than seeking to emulate an LMS, iQualify began 
from the paradigm of creating a learning experience platform. The starting point of development 
was the user experience (UX) of the online student. 

A user experience includes “the individual’s entire interaction with the thing, as well as the 
thoughts, feelings, and perceptions that result from that interaction” (Tullis & Albert, 2013, sec. 
1.1, para.4). The UX approach considered how students use contemporary internet-based 
applications and on-screen resources. Online services such as Facebook, the on-screen functions 
of such applications as Kindle Reader, and device independence, provided examples of how a 
user’s experience with online materials might be enhanced. The UX aspects of the iQualify 
system include providing point-of-interest discussion tools rather than requiring students to go to 
a forum tool and then return; note-taking that is facilitated in the online system; removing all 
clutter to provide a clean on-screen page of content; out-of-the-way navigation that can be 
activated when required; and optimising the synchronisation of student place and presentation 
across devices. A suite of interactive, formative activity options is also available, as is a series of 
analytics data. Figure 1 shows the indications of student progress in iQualify. 
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Open Polytechnic 



My Courses 


Current courses 



Figure 1 My Courses page in iQualify, showing progress indicators 
© Open Polytechnic 

The study of online learning from a UX perspective is overdue. Previous studies have 
investigated LMSs “in terms of usability, accessibility, instructions for use, navigational 
facilities, and content” (Byrne & Bates, 2009, p. 131), but comparisons across systems in 
literature are yet to reflect a UX perspective (Zaharias & Mehelenbacher, 2012). In education, the 
overall UX of the student is a combination of the elements of technology, pedagogy, and the 
actual learning that takes place. All three are interdependent. The student’s UX consists of both a 
different online experience as the result of a new platform, and a new approach toward 
courseware development. Evaluative LMS studies also tend to consider the LMS from the 
perspective of supporting on-campus students (Gavira & Omoteso, 2013; Volery & Lord, 2000). 

In developing and adopting iQualify as its platform for the online student experience, Open 
Polytechnic made a deliberate decision to develop online-only courses and to extend its resource- 
based approach. Students are not provided with any print materials (although an option for 
providing printed readings - not materials written for iQualify pages - is being developed). Nor 
are students required to purchase a set text. This decision to go online-only for course 
development was deliberate, and represented a further institutional departure from using Moodle 
for online learning. Materials are purposefully prepared to leverage the possibilities of a digitally 
enriched on-screen experience. 

There are several reasons for discouraging print. The first is that not having a print option means 
Open Polytechnic is free to develop enriched online materials that use embedded media and 
activities. If students get print copy, they tend to not benefit from these activities. Online study 
also enables students to benefit from writing and engaging with social notes. Further, when 
students engage with online materials, they generate important and detailed analytics data that 
are then used to improve student support and the online course materials. Students can print each 
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online page but there is no easy way to aggregate pages for printing. Figure 2 shows a text-only 
iQualify page of learning materials. 

Open Polytechnic 


Leadership (*) Mmins 

Defining leadership 


A leader is best when people barely know that he exists, not so good when people obey and 
acclaim him, worst when they despise him. Fail to honor people, they fail to honor you. But 
of a good leader, who talks little, when his work is done, his aims fulfilled, they will all say. 
'We did this ourselves." 

(Laozi. the founder of philosophical Taoism, from the Tao Te Ching) 



What is 'leadership? This seems a pretty basic question but actually defining leadership qn 

continues to create discussion between academics and lay people alike. In fact, Stogdill 
(1974) once estimated that there were approximately 35 000 definitions of leadership in 
academic literature. 

The word 'leader' is derived from the Anglo-Saxon laedere. which means to take people _ 
on a journey'. Using this concept, one definition of a leader is: 


... someone who sets ideas, people, organisations and societies in motion, someone who 
takes the world of ideas, people, commerce, organisations and societies on a journey. To 
lead such a journey requires vision, courage and influence (Adler, 2006) 


Figure 2 An iQualify page (without activities or embedded media) 

© Open Polytechnic 

The course design approach used for iQualify consists of revising print materials so they are not 
aligned to a set text and are optimised for online-only learning. Video clips, engagement 
activities and PDF readings are embedded in materials written for on-screen reading. Most 
modules also have a final self-marking quiz to help students to consolidate their learning. 

The iQualify platform was developed using an agile architecture (Schwaber & Beedle, 2001; 
Stellman & Greene, 2013). This means that new features are added frequently, and a minimal 
viable product forms the criteria for feature implementation. In the first trimester of 2015, 
iQualify courses were available only via a browser, and were not available offline. Further, full 
integration with Open Polytechnic systems was incomplete. This meant that, for example, 
students had to log into the iQualify platform as well as their own online Open Polytechnic page. 
Because it was designed to provide an intuitive experience, students were not given any 
orientation. 

Open Polytechnic opened iQualify to students in the first trimester of 2015. In the latter half of 
2014, six Bachelor of Business courses were especially adapted to be set-text-free and online- 
only through the iQualify platform. The course subjects included accounting, business 
communication, business computing, ethics, financial management, and leadership. Four were 
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first-year degree courses, with the balance being second-year degree courses. Open Polytechnic 
therefore used two different online platforms to deliver Bachelor of Business courses, though all 
of those designed for iQualify were offered only in the new system. 

A thorough and multi-faceted comparison and evaluation was conducted. The objectives of the 
exercise were to provide evidence of the effectiveness of the online-only approach, the efficacy 
of iQualify as an online learning experience platform, and the student perception of an online- 
only learning approach. 

Comparison and evaluation method 

A multi-dimensional approach was applied for the purposes of comparison. Of interest to the 
polytechnic were the student enrolment behaviour profile and outcomes (explained below), 
course evaluations, and direct student perceptions of Moodle and iQualify from a UX 
perspective. 

The student enrolment behaviour profile considers course withdrawal and completion through 
2014, when the courses were offered only in Moodle, and 2015 when the online-only, textbook- 
independent versions of the courses were offered only in iQualify. With one exception, all of the 
courses under consideration were offered to students in the first trimester of both 2014 and 2015 
(the exception, offered only in the latter part of 2014, is indicated in the findings). The purpose of 
comparing 2014 and 2015 student enrolment behaviour was to determine whether removing set 
texts and moving to an online-only course significantly affected student behaviour. In particular, 
withdrawal behaviour might demonstrate a negative student response to an online-only study 
environment, and fewer students succeeding could indicate a flawed design. 

Course evaluations for 2014 and 2015 were also compared. Open Polytechnic uses an online 
survey in two parts for course evaluations. Initially, students were asked for their overall 
impression of the course, which they scored on a scale from 1 to 5. They were then given the 
option of committing to a further set of questions that provided more detail of their experiences. 
The purpose of the comparison was to determine whether the iQualify experience significantly 
shifted students’ overall appreciation of the course. 

Finally, Level 5 (first-year-degree equivalent) business students who had enrolled for the first 
trimester of 2015 were invited to participate in an online survey. The population for this survey 
excluded those students who enrolled in the two Level 6 (second-year-degree equivalent) courses 
added to the iQualify system. The survey was available in Survey Monkey, 2 and students were 
invited to participate by email in a 1-week timeframe. Two reminders were sent. 

Three discrete groups were invited to participate in the survey: 

1. Students using only iQualify. The population of this group was 38, of whom 13 
responded (a 34% response rate). 

2. Students using iQualify and Moodle at the same time. The population of this group was 
33, of whom 5 responded (a 15% response rate). 

3. Students using iQualify who have previously studied using Moodle. The population of 
this group was 188, of whom 78 responded (a 41.5% response rate). 

This analysis focuses solely on Groups 1 and 3. The response rate of Group 2 was regarded as 
being too small to enable any meaningful analysis. 


2 http://www.surveymonkey.com 
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The survey instrument combined measures from several studies (Byrne & Bates, 2009; Gavira & 
Omoteso, 2013; Park, 2014; Tullis & Albert, 2013, focusing on the UX and online learning 
experience during students’ study (see the appendix to this paper). The survey asked students to 
rate a series of factors using a standard Likert scale of 1 (poor) to 5 (excellent). The overall scales 
of usability, accessibility, instructions for use, navigational facilities, and content were measured, 
in addition to an overall rating. 

Group 1 respondents were asked to rate only iQualify, while respondents from Groups 2 and 3 
were asked to rate both iQualify and Moodle. 

The online survey also invited students to make open-ended comments relating to their 
perceptions of each system, and they had the opportunity to participate in a telephone interview. 
Seven students, all from Group 3, were prepared to be interviewed. The purpose of the interview 
was to explore the student experience of both systems in more detail. 

It is important to note that, while “content” is a specific scale of the instrument, respondents are 
likely to have considered course materials across most measures. Separating the technology from 
how materials are made available is not an objective of this study. Rather, findings must be 
understood in the context of a new online learning platform that also happens to make a new 
form of online course available to students. 

Findings 

Findings from each of the comparison and evaluation exercises follow. 

Comparison of student enrolment behaviour and outcomes 

Enrolment and completion data for each course were checked to see whether adopting iQualify 
and online-only delivery influenced student withdrawals and success. This data is based on the 
student population of those enrolled in the courses of interest in 2014 and 2015. Table 1 
summarises the data for the first trimesters of 2014 and 2015 with the exception of Course 6, 
which was not offered in the first trimester of 2014 (the second trimester of 2014 is used as a 
comparison instead). The only changes between the 2014 and 2015 course offerings were the 
redevelopment of courses for the iQualify system, and the iQualify experience. 


Table 1 Student participation behaviour in Moodle and iQualify courses, 2014 and 2015 



Withdrawn (total) 1 

Not successful 

Successful 



2014 

2015 

2014 

2015 

2014 

2015 

X 2 

P 

Course 1 

59 

74 

20 

30 

103 

94 

3.44 

0.179 

Course 2 

0 

6 

0 

2 

6 

18 

2.46 

0.292 

Course 3 

14 

8 

4 

0 

55 

30 

2.17 

0.338 

Course 4 

3 

7 

4 

9 

10 

18 

0.16 

0.923 

Course 5 

14 

21 

6 

3 

54 

59 

2.11 

0.348 

Course 6 

9 

3 

12 

16 

59 

56 

3.49 

0.174 


1 This percentage includes students who withdrew before the refund period; that is, they enrolled but withdrew within 
4 weeks of the start of the course. 


Chi-square analysis was applied to each course to test for significant differences between the 
2014 and 2015 course offerings. No significant differences were found for any course between 
2014 and 2015, at p < 0.05. It appears that adopting iQualify has not made any significant 
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difference to student withdrawal patterns or success, despite the deliberate shift to an online-only 
study paradigm with no print resources provided. 

Comparison of course evaluations 

Open Polytechnic provides an online interface for students to provide course feedback after they 
complete each trimester. The evaluation survey begins with a request for students to provide an 
overall rating for their course from 1 (very poor) to 5 (excellent), before giving students the 
option to proceed to more detailed questions. Table 2 shows the mean overall ratings by course. 

Table 2 Overall student evaluations for Moodle and iQualify courses, 2014 to 2015 



Mean (s.d.) overall rating 

Number of responses 


2014 

2015 

2014 

2015 

Course 1 

4.2 (0.75) 

4.2 (0.86) 

54 

26 

Course 2 

4.1 (0.5) 

4.5 (0.58) 

4 

4 

Course 3 

4.3 (0.91) 

3.5 (1.04) 

36 

11 

Course 4 

4.1 (0.69) 

4.4 (1.00) 

7 

3 

Course 5 

4.2 (0.86) 

4.2 (0.85) 

67 

34 

Course 6 

3.2 (0.94) 

3.4 (1.10) 

47 

20 


The mean overall rating for each course was similar but slightly higher in the second year, with 
the exception of Course 3, for which the rating dropped. A Mann-Whitney U-test confirms that 
the results for 2014 and 2015 are not significantly different, at p <= 0.05. There is no significant 
difference through 2014 and 2015, even with the Course 3 outlier included. Course 3 received 
several evaluative comments that related to the quality of course video clips, a desire for more 
worked examples, and requests for printed notes. The desire for printed notes was not unique to 
Course 3; however, the technical nature of the course may have accentuated the feedback. 

Student comments across all courses indicated a polarisation of preference for printed materials. 
Some students (n=10) expressed concern that materials could not be downloaded or easily 
printed for revision; others commented positively about the flexibility that online-only and 
device-synchronised material provided («=5). However, most student comments did not mention 
the paper or online format in particular. 

Comparison of student UX 

The student experience of iQualify consisted of an evaluation survey based on UX characteristics 
and follow-up interviews. Two user groups—1 and 3—returned sufficient data for analysis. 
Group 1 comprised Level 5 students who used iQualify but had not used Moodle. Typically these 
students were new to Open Polytechnic. The online survey for Group 1 (2V= 13) provided the 
results shown in Table 3. 
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Table 3 UX survey results for iQualify-only students (Group 1) 



Mean 

(iQualify) 

Standard Dev 

(iQualify) 

Mode 

(iQualify) 

Scale 

Usability 

3.53 

0.98 

4 

Accessibility 

3.82 

1.01 

4 

Instructions for use 

3.41 

1.05 

4 

Navigational facilities 

3.88 

0.96 

4 

Content 

3.61 

0.9 

4 

Overall 

3.64 

1.03 

4 


Respondents to the online survey in Group 1 commented on several improvements that would 
improve the iQualify system. Most comments related to wanting offline access, single sign-on, 
email notification of changes, and a search function. Respondents also commented that 
portability, contents (navigation), access, and peer engagement were useful features, as was the 
fact that iQualify “retained where you left off’. 

Group 3 respondents were Level 5 students who used iQualify in the first trimester of 2015 and 
had previously used Moodle. This group was asked to comparatively rank iQualify and Moodle 
using the survey instrument. Table 4 shows the results for Group 3 (N= 78). 


Table 4 UX survey results comparing students with Moodle and iQualify experience (Group 3) 



Mean 

Standard Dev 

Mode 

x 2 

Scale 

Moodle 

iQualify 

Moodle 

iQualify 

Moodle 

iQualify 

P 

Usability 

3.54 

3.76 

1.06 

1.11 

4 

4 

0.036 

Accessibility 

4.15 

4.03 

0.97 

1.14 

5 

5 

0.3639 

Instructions 
for use 

3.81 

3.93 

0.99 

1.14 

4 

5 

0.328 

Navigational 

facilities 

3.52 

4.02 

1.07 

1.14 

4 

5 

3.6E-06 

Content 

3.58 

3.85 

1.02 

1.08 

4 

4 

0.007 

Overall 

3.55 

3.87 

1.01 

1.16 

4 

4 

0.0146 


Respondents in Group 3 rated iQualify more favourably than those in Group 1—this may have 
been because Group 3 was asked to compare two systems. Significant differences in favour of 
iQualify over Moodle are apparent in the usability, navigational facilities, and content scales. 
Overall, iQualify was perceived to be significantly better than Moodle, at p < 0.05. 

Respondent comments from Group 3 provided additional insight into perceptions across the two 
systems. Fifteen of the 61 respondents who provided written feedback commented on the need 
for printed materials or PDF versions of all courseware. Most of these respondents indicated a 
preference for the latter. Comments such as “I did not like the fact that PDF copies of the 
learning material were not available, these are handy to have when travelling”; “I would rather 
have hard copy records as well”; and “[I would like the] ability to download notes as I have 
limited internet access” were typical. In contrast, several Group 3 respondents expressed 
appreciation for the print-free format. Statements such as “Everything is online [so there are] no 
papers to wor[ry] about”; “All information I needed was online (no textbooks)”; and “Not 
wasting trees to print manuals was pretty cool... It is a step in the right direction as far as 
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paperless learning is concerned” were all clear endorsements of the online-only approach. The 
online-only format was also appreciated by the respondent who remarked “I’m able to study on 
the bus to and from work, making use of time otherwise wasted ... I really appreciate being able 
to study on my tablet, rather than lumping around textbooks and modules”. Videos and 
engagement exercises were also identified as contributing to the iQualify experience. 

Most student comments expressing a preference of one system over the other (N= 21) preferred 
iQualify (V=17), with statements such as “use iQualify more”; “I wouldn’t change anything but I 
preferred iQualify”; and “I enjoy using iQualify as it is” being representative. One student 
commented that Moodle was “a very old, very unuser [sic] friendly system that needs replacing 
entirely”. As in Group 1, many Group 3 respondents commented on the need for an additional 
log-in step to access their iQualify courses, and the benefits of email notification to updates. 

Only one respondent indicated a preference for Moodle, commenting that using iQualify “was 
such a frustrating process. I prefer [Mjoodle and would try to avoid papers that [use 
iQualify] ... it takes [a lot] of getting used to”. One stated that “[I] haven[’]t enjoyed either 
system”. 

Comments from Group 3 respondents were similar to those from Group 1 in that they were 
especially appreciative of iQualify navigation, memory of last page visited, and engagement 
activities. The layout of materials in iQualify also attracted comment, with one respondent 
simply remarking that “it is better than [Mjoodle”. Other appreciative comments related to 
iQualify’s ease of use, progress tracker, and attractive layout. 

Student interviews 

Seven respondents from Group 3 were interviewed to get first-hand accounts of using Moodle 
and iQualify. An important context for all student responses is their appreciation of flexible 
access to study. Most respondents studied early in the morning or late at night, and between work 
and family commitments. The design of iQualify courses recognises such study habits, though 
not all appreciated the page set-up used by iQualify (Respondent 7’s comments on speed reading, 
and the need for those who prefer hard copy to print multiple pages, are mentioned below). Two 
respondents indicated that their desktop or laptop computer was not the ideal tool for their study, 
but others appreciated the flexibility of study through their mobile devices. 

Of the seven interviewees, four mentioned the lack of a print option. Respondents had a definite 
preference for printed readings (in the form of articles), though the materials written directly into 
iQualify were also mentioned. Respondent l’s comments are representative: 

. . . my actual preference is Moodle because you can print out modules ... I don’t actually 
like sitting down at a computer. I just print off the modules. I read the module in my own 
time. With iQualify you can only print page by page, the page you are currently on, which is 
a bit of an annoyance. (Respondent 1) 

This same respondent’s preferred means of engaging with online materials is to “print [the 
material] out, I’ll take away, read it, if there’s something I need to do online, I’ll log on, do what 
I need to do online, get off-line, go back to my printed materials”. Respondent 7 added that it 
was more difficult to speed read on the screen. Respondent 1 did close with an indication of 
changed behaviour: 

If I had the choice I’d go for Moodle for that printing, and that’s my biggest thing. I found it 
an annoyance when IQualify came out and I couldn’t print out chapters, but I’m used to it 
now as I’m on another course with IQualify and I know what to expect. . . I just have to 
change the way I learn I guess ... I guess I’m pretty old school. (Respondent 1) 
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The perception of being ‘old school’ was also mentioned by Respondent 4, who remarked that 
“sometimes I just wish this was on paper”. Paper was preferred by Respondent 7 because paper 
provided the opportunity to “flick and find [information] a lot quicker”. Respondent 7 added “it 
probably is an age related thing as well, I’m more comfortable with flicking through a book, 
backwards and forwards”. Respondent 6 was an exception, stating that “I preferred the online 
version rather than the printed version”. Having no way to highlight text was also raised as a 
technical issue with iQualify materials, though Respondent 6 mentioned “I typed or cut and 
pasted things, that made it easier . . . pieces of text or quotes, or whatever”. 

Poor internet connectivity also affected three respondents, though all were philosophical about it. 
Respondent 1 stated that “There are times when I might not have a good connection or I’m at 
work, I might not be at home, I don’t have an internet connection . . . But when you’re on it, and 
you’ve got a good connection, it’s great”. Another respondent remarked that “I can remember 
there was [sic] some technical difficulties, which was obviously because maybe it was in its early 
stages”. 

All respondents remarked positively on the aesthetics of iQualify and the design of the courses 
they encountered. Respondent 1 commented that they preferred the “look and feel” of iQualify, 
though overall preference was for a combination of both that included access to printed 
materials. iQualify’s modular design of courseware, percentage of study completed, and ‘wrap- 
up’ quizzes at the end of modules were particularly popular. Respondents indicated that email 
notifications for new discussion contributions, an index of readings, and a search feature would 
be useful. Comments about course design included appreciation of the use of video (with some 
exceptions relating to specific use of third-party clips), but one respondent indicated videos were 
an inefficient way to learn. 

Three respondents also reflected on the use of forums in iQualify. Respondent 5 said that “I find 
the whole student online forums a bit frustrating really . . . ’Cause I think there’s a lot of rubbish 
on them [laughs]. . .1 don’t have much confidence when it comes to contributing my own stuff’. 
Respondent 7 added that “I didn’t find screeds and screeds of pages of discussion helpful, 
because everyone had different ideas and that became confusing because you couldn’t actually 
brainstorm and feedback straight away”. Respondent 6 said: 

... at first the lecturer actually summarised [student views], which I thought was really 
useful. But as time went on I think that got less and less which meant you sort of had to 
plough through everybody’s different experiences and different views which could become a 
little bit tedious. And also repetitive, because people often had the same ideas. But yeah, I 
did like that and the fact that everyone was contributing. (Respondent 6) 

Overall, all respondents were satisfied with their online learning experience with Open 
Polytechnic through iQualify. A statement from Respondent 5 is representative: 

I’ve really liked it and I’m probably in the age group where we find technology a little bit 
challenging sometimes. So I was a bit surprised really because there was not much of a 
warning that it was going to be different. But, I have to say, generally speaking, I really 
enjoyed it. (Respondent 5) 

Respondent 5 also appreciated that iQualify gave them more confidence in using technology, an 
implicit benefit associated with 21st century literacies. 

Discussion 

This study provides evidence that a well-designed, online-only study experience can be 
equivalent in performance, and better by way of UX, than courses that combine print materials, 
set texts, and online forums. The findings provide evidence to support an online-only, iQualify- 
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based education from the perspective of the student UX compared with Moodle, though with 
some reservations regarding access to printed materials. Student enrolment behaviour and 
outcomes, and course evaluations, remained similar for both systems. However, students 
reported a better UX when studying with the iQualify platform. 

The two UX factors for which iQualify was not perceived to be significantly better than 
Moodle—accessibility and instructions for use—need some comment. Single sign-on (SSO) is 
under development for the Open Polytechnic implementation of iQualify, and this will address 
most of the student comments related to accessing the iQualify system. Instructions for use are 
also being considered, particularly as some students reported difficulty in making the transition 
to online-only study. An overview of iQualify’s features, including device synchronisation, 
strategies for on-screen reading, and advice for note taking, will probably give iQualify learners 
more confidence in their use of the platform. 

The findings indicate that students are willing to make, and are capable of making, a transition to 
an online-only study environment. Although some students expressed a strong preference for 
print, others appreciated the benefits of portability and ready access provided by online materials. 
Student enrolment behaviour and outcomes findings demonstrate that the shift to an online-only 
experience did not result in a significant increase of student withdrawals or failure. It seems that, 
while many students might prefer printed materials, having them is not necessary for academic 
success, and students perceive the shift towards online-only education to be a progressive one. 
Students reported their own strategies for printing materials when they felt print was required. It 
is anticipated that the development of an offline app version of iQualify, which will enable a 
download of all course materials onto a mobile device that will synchronise when online access 
is available, will be well received. This will transform iQualify from an online-only experience to 
an on-screen-only experience. 

It is also clear that the UX of iQualify as a platform requires consideration alongside the student 
experience with the course materials. It is not sufficient to consider an online system solely on 
the basis of its features, because these cannot be fully separated from the design of the online 
courseware. The development of iQualify assumes a particular approach to course development, 
whereby course designers embed media and engagement activities in the learning materials, 
resulting in an integrated online study experience. Importantly for the findings, this indicates that 
Group 3 respondents were, effectively, comparing textbook-assisted print materials and a 
complementary Moodle experience, with an online-only and enhanced iQualify experience. 
Although respondents to the survey were asked to compare their user experience of the online 
systems, their evaluations combined the features of the actual system and the course dynamics. 
Evaluation of any online platform should consider learning materials alongside actual platform 
features as part of the overall UX. Comparing the same online-only and enhanced learning 
materials across iQualify and Moodle was beyond the scope of this evaluation. However, the 
online experience that specific features of iQualify as a platform was designed to provide gave 
Open Polytechnic confidence to develop online-only courses. 

iQualify is still in the early stages of development, and its analytics tools are not yet fully 
matured. Once in place, analytics data will provide an evidence base of student engagement and 
progress through materials (de Freitas et al., 2015; Ferguson et al., 2014). They will also enrich 
student support services and courseware development. Tutors can be alerted to students who 
could benefit from a motivating phone call; students who have not logged in across a certain 
timeframe can be automatically contacted; and engagement activities can be checked to see 
whether students are providing the sorts of responses that were anticipated, or if the activities are 
being used at all. Analytics will provide an evidence base of student engagement and 
effectiveness of materials. It is anticipated that these developments will further enrich the student 
experience and lead to stronger student outcomes. 
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Because iQualify was developed in-house using an agile project management approach, it will be 
possible to enrich the platform with new and improved features while it is being used. 
Evaluations and comparisons such as this provide key mechanisms for further enhancing the 
iQualify system. 

Conclusion 

A UX comparison of iQualify and Moodle—the former featuring an online-only and textbook- 
independent course design—demonstrates a significantly better UX across key criteria (usability, 
navigational facilities, content and overall) in favour of iQualify. Although student enrolment 
behaviour and outcomes and overall course satisfaction remain comparable, improvement in the 
usability, navigational facilities, and content factors of the user experience show that Open 
Polytechnic’s move to a new online learning platform has made a positive difference for 
students. 

A UX-based approach to online platform comparison has demonstrated an effective means of 
investigating student perspectives of online learning. A UX approach transcends the comparison 
of features and course evaluations, both of which determine how students engage with online 
learning. Further studies investigating the user experience are likely to add considerably to 
effective online education practice. 

Finally, it is clear that any UX comparison across different online learning platforms must 
include consideration of learning materials. This study does not compare the same course 
materials across Moodle and iQualify. Results show that iQualify provides an improved UX for 
students although it is not possible, from the data, to determine the contribution made by 
specially prepared courseware. 
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Appendix: User experience rating factors 

Usability 

• The online system was enjoyable to use. 

• The online system complemented the course materials. 

• The online system worked in the way I would have expected it to. 

Accessibility 

• I was able to access my learning when I needed to. 

• The online system was convenient to log in to. 

Instructions for use 

• It was clear how to use the online system. 

• I think I used the online system well. 

Navigational facilities 

• The online system was easy to work with. 

• In the online system it was easy to pick up again from where I left off. 

• I could easily find my way around the online system. 

Content 

• The structure of materials in the system added to my learning. 

• The online system itself helped me to achieve my learning goals. 

• The learning materials in the system contained useful activities. 

Overall 

• Overall, how would you rate your learning experience through the online system? 
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